PREVENliWT 

UPDATES 

February 2006 


Evaluating Environmental Management 
Approaches to Alcohol and Other Drug 

Abuse Prevention by William Dejong, Ph.D. and Linda M. Langford, Sc.D. 


InR Recent years have seen an upsurge in 
prevention work focused on changing 
the campus and community environments in 
which college students make decisions about 
alcohol and other drug (AOD) use. This 
approach, called environmental manage- 
ment, is based on three fundamental premises: 

1. Substance use problems are aggravated by a 
physical, social, economic, and legal envi- 
ronment that increases both the availability 
and appeal of alcohol and other drugs. 1 

2. The most cost-effective means of reducing 
the number of AOD problems that a commu- 
nity experiences is to change that environ- 
ment, thereby increasing motivation to avoid 
illegal or excessive substance use and its 
negative consequences. 2 

3. Successful environmental management 
requires a coalition of campus and commu- 
nity officials to develop and execute a 
strategic plan that features science-based 
programs and policies. 3 

Environmental management strategies can 
be organized into five major categories: (1) 
offering alcohol-free social, extracurricular, 
and public service options; (2) creating a 
health-promoting normative environment; (3) 
limiting alcohol availability; (4) restricting the 
marketing and promotion of alcohol; and (5) 
creating and increasing enforcement of policies 
and laws. All five strategies involve a wide 
range of possible program and policy options 
(see sidebar “Five Strategic Objectives Focused 
on Environmental Change”). 4 A comprehen- 
sive program will include multiple interven- 
tions targeted to local problems. 

Health educators and counselors who run 
traditional education and treatment programs 
work to persuade individual students to lower 
their substance use, but such change is much 


more difficult when the campus 
and community environments 
continue to facilitate or even 
encourage that high-risk 
behavior. Successful prevention 
requires a wide-ranging 
approach that has environ- 
mental management as its 
foundation. 

Conscientious academic 
administrators who apply 
environmental prevention 
approaches will want to 
know if their efforts are 
effectively reducing alcohol 
and other drug problems 
(see sidebar “Principles of 
Effectiveness for 
Prevention Programs” on 
p. 2). This requires 
systematic evaluation — 
a process for collecting, 
analyzing, and 
reporting information 
that can answer that 
question. This publi- 
cation is designed to 
provide interested 
college and univer- 
sity administrators 
with a brief orienta- 
tion to the evaluation 
of prevention programs that 
are based on the environmental management 
approach. 

In practice, campus officials who have applied 
an environmental management approach to 
AOD abuse prevention have found it useful to 
view program development and evaluation as 
an iterative process, with evaluation findings 
helping to inform program modifications. 
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The evaluation process has five steps: 

(1) describing the intervention; (2) identifying 
process measures; (3) identifying outcome 
measures; (4) selecting a research design; and 
(5) utilizing the results. 
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Describing the Intervention 

A strategic plan based on the environ- 
mental management approach usually entails 
several strategies that may evolve and change 
over time. Therefore, an essential first step in 
the evaluation is to create a summary of the 
prevention effort by developing a logic model or 
evaluation map. The puipose of the logic model 
is to list the program’s key components and 
then outline how each one is supposed to 
work — that is, to show the chain of events that 
is anticipated, moving from specific strategies 
and activities to intermediate and long-term 
outcomes. Figure 1 (see p. 5) is a logic model 
for a responsible beverage service (RBS) 
program, an environmental strategy designed 
primarily to prevent excessive consumption of, 
and reduce minors’ access to, alcohol. 

The logic model serves as the basis for the 
evaluation plan, noting both the interim steps 
and outcomes that can be measured and 
assessed. If an evaluation were to show that a 
program’s goals and objectives were not 
achieved, staff could ascertain where in the 
logic model’s expected sequence an activity did 
not work as planned (e.g., high turnover 
resulting in some staff not being trained in 
RBS). If the expected outcomes were obtained, 
having an evaluation that examined each step 
in the logic model would help to establish that 
the activity contributed directly to achieving the 
program’s goals and objectives. 

It is important that an evaluator be part of 
the planning process at this early stage. 5 The 
evaluator may help to ensure that: (1) the 
needs assessment is well designed and can 
provide baseline (i.e., pre-intervention) data for 
the evaluation; (2) the intervention plan 
features policies and programs with demon- 
strated effectiveness or a solid foundation in 
behavior change theory; (3) the program’s 
goals and objectives are precisely stated so that 
measurable outcomes can be specified; (4) each 
program component and policy can be linked 
logically to specific objectives; and (5) adequate 
resources are in place to ensure full and 
adequate implementation of the program plan. 

Identifying Process Measures 

The second step in the evaluation 
process is to develop process measures for docu- 
menting the nature, extent, and quality of 


program implementation. Stated simply, process 
measures are used to assess whether the preven- 
tion effort is being implemented as designed. 

This infonuation is critical. Once the inter- 
vention is under way, process data may be used 
to monitor progress and determine whether 
corrective steps are needed. If a particular inter- 
vention were to fail, having the process data 
would make it possible for evaluators to deter- 
mine whether that failure might have been due 
to inadequate implementation. 

One group of process measures looks at 
utilized resources — that is, the staff, volunteers, 
and funds used to operate the program, which 
might differ from what was originally planned. 
Typical measures include a count of full-time 
equivalents (FTEs) and expenditures. 

Another group of process measures looks at 
activity levels and the numbers of people or 
groups being served by the program. For an 
RBS program, measures include the number of 
bars and restaurants signing up for the RBS 
program, the number of staff and managers 
receiving training, and the number of random 
compliance checks and other enforcement 
activities. 

A key process in environmental management 
is the fonnation and operation of a campus and 
community coalition. 6 Certain process indica- 
tors are linked to the activities of the coalition 
itself — establishing a core leadership team and 
a subcommittee structure; developing rules of 


governance; recruiting membership; 
conducting a needs assessment, including a 
scan of the campus and community environ- 
ment; identifying science-based program and 
policy options; outlining a strategic plan, 
including goals, objectives, environmental 
management strategies and tactics; and imple- 
menting that plan. 

Also critical are process measures of coalition 
function — member commitment to the coali- 
tion’s goals; degree and quality of participation; 
perceived support from high-level campus and 
community officials; and degree of satisfaction 
with the coalition’s leadership and structure, 
the conduct of meetings and other activities, 
communication and outreach, and methods for 
decisionmaking and conflict resolution. The 
most important question, however, is whether 
the structure and function of the coalition 
helps campus and community officials to 
achieve the environmental changes specified in 
the strategic plan. 

Identifying Outcome 
Measures 

Outcome measures to assess a program’s 
success or failure also need to be identified. 
There are two types of outcomes to consider: 
changes in behavior, and changes in structure 
or functioning of the environment. 
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Behavioral Outcomes 

When specifying behavioral goals and objec- 
tives, it is important to differentiate between 
intermediate and long-term outcomes (see 
fig. 1). 

The prevention program’s logic model will 
specify a set of intermediate outcomes that are 
necessary for achieving the desired long-term 
outcomes. For an RBS program, for example, 
intermediate outcomes include several staff 
behaviors outlined in the RBS protocol — 
refusing entrance to minors or intoxicated 
patrons, refusing service to patrons who could 
become intoxicated, arranging alternative trans- 
portation for patrons who should not drive, and 
providing free food and soft drinks for desig- 
nated drivers. 

Looking at long-term outcomes, evaluation 
plans usually include measures of student 
behavior, because the ultimate goal of any 
prevention program is to reduce AOD problems 
among students, including illegal alcohol and 
other drug use. A typical RBS program assesses 
the purchase and consumption of alcohol and 
the use of alternative rides or designated drivers 
to avoid driving after drinking. 

The best method for assessing student 
behavior is a survey administered to a randomly 
drawn sample of students. Self-reports, when 
provided under conditions of confidentiality, are 
known to produce population-level estimates 
that are generally both valid and reliable. 7 
Pencil-and-paper surveys work, but Web-based 
surveys, with participants recruited by e-mail, 
are increasingly the method of choice due to 
lower costs and the ease and speed of adminis- 
tration. These two methods generally produce 
comparable response rates. 8 Existing surveys 
may be examined for questions to include. 9 

To create the study sample, most college 
registrars are able to generate a randomly 
chosen list of enrolled undergraduates. In 
general, administrators should send surveys to a 
sample of between 400 and 1,000 students. If 
the school's budget permits it, a sample of 
1,000 is far superior. Even under the severest 
budget constraints, the sample should not be 
any smaller than 200. 10 

Essential to a successful survey is a high 
response rate. While researchers generally want 
to achieve response rates of 70 percent or 
higher, recent national surveys have typically 
reached rates of between 50 and 60 percent. 11 


Factors that may increase the response rate 
include procedures to protect student confiden- 
tiality, pre-survey messages notifying students 
about the survey, sending multiple reminders, 
and offering financial or other incentives for 
completing and returning the survey. Response 
rates also tend to be higher with shorter 
surveys. 12 

Student behavior also may be assessed by 
structured field observations of student behavior. 
Measures might include numbers of student 
patrons at alcohol-free venues or numbers of 
students who are carrying drinks in public. 
Structured field observations also may be used 
to assess student intoxication by measuring 
blood alcohol concentration (BAC) with a 
passive breathalyzer. Unobtrusive measures also 
are possible — for example, tire volume of alcohol- 
related litter (cans, bottles, drinking cups). 

Outcome measures also may include AOD 
consequences. Surveys may ask students directly 
about consequences resulting either from their 
own or other students’ AOD use. Information 
from various archival records may be useful, 
too — for example, the number of students seen 
in local emergency rooms or the student health 
center for alcohol-related injuries or illnesses, 
number of students arrested for AOD-related 
infractions, building and equipment damage 
reports, and number of residence hall 
complaints due to AOD-related behavior. How 
well these records are kept might change over 
time, which could make these data difficult to 
interpret. 

Environmental Outcomes 

An evaluation of programs focused on environ- 
mental change should not be restricted to 
measures of student behavior and AOD-related 
consequences. Seeing meaningful change at this 
level might require several years, and there are 
many other factors, beyond the control of 
campus and community officials, that might 
influence individual outcomes (e.g., changes in 
alcohol excise taxes, new alcohol products, 
greater availability of low-priced illicit drugs) . 

Also important are measures of the environ- 
ment itself. 13 Measuring the environment as 
part of needs assessment and evaluation has 
two key benefits — identifying and raising 
awareness of environmental factors that 
contribute to AOD problems, and documenting 
intermediate outcomes of environmental 


change efforts that set the stage for changes in 
individual student behavior. 

There are two primary categories for environ- 
mental measurement: (1) self-reports; and (2) 
archival records and program documents. Self- 
reports may come from suiveys, interviews, or 
focus groups with students, faculty, community 
residents, and others. Written records also are 
an important data source. On most campuses, 
the evaluation team will need to work with the 
various campus departments (e.g., campus 
police, student health seivices) and community 
agencies (e.g., local police, hospital emergency 
room) to develop forms and recordkeeping 
procedures to improve the quality of informa- 
tion received. 

Measures should be chosen that correspond 
to the particular interventions being employed 
and the specific environmental changes that are 
outlined in the logic model. Such measures, 
organized by environmental management 
strategy, include the following: 

Alcohol-Free Options 

Student Self-Report: (1) percentage reporting 
attendance at substance-free social events; (2) 
number of hours spent in volunteer public service. 

Archival Records: (1) number of student organi- 
zations given funds to support substance-free 
events; (2) number of hits on a college-sponsored 
Web site listing recreational alternatives in the 
community. 

Normative Environment 

Student Self-Report: (1) reported exposure to 
messages about true drinking norms on 
campus, designed to correct existing mispercep- 
tions; (2) reports of mixed messages about AOD 
use from faculty, resident assistants, and other 
staff. 

Archival Records: (1) number of beds and the 
occupancy rate for substance-free housing on 
campus; (2) percentage of academic courses 
holding Friday classes. 

Alcohol Availability 

Student Self-Report: (1) perceived ease of 
acquiring alcohol from social and commercial 
sources; (2) average price paid for standard 
alcohol products (e.g., six-pack of beer) . 

Continued on page 4 
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Archival Records: (1) number of liquor licenses 
within one mile of campus; (2) number of bars 
and restaurants promoting “happy hours” in 
campus newspaper advertisements. 

Alcohol Marketing and Promotion 

Student Self-Report: (1) reported exposure to 
party announcements that mention drinking; 
(2) reported exposure to bar advertisements in 
campus and local newspapers. 

Archival Records: (1) number of alcohol adver- 
tisements per week appearing in the student 
newspaper; (2) number of alcohol industry- 
sponsored events at local bars and restaurants. 

Policy Development and Enforcement 

Student Self-Report: (1) perceived risk of arrest 
for driving under the influence (DUI); (2) reported 
exposure to information about the state’s “zero 
tolerance” law for underage students. 

Archival Records: (1) number of special opera- 
tions to enforce the minimum-age drinking 
law; (2) number of students cited for AOD- 
related conduct violations. 

In some cases, systematic field observations 
also may be used to assess environmental 
change. Indicators include the number of kiosk 
messages that promote high-risk drinking, the 
number of price discounts being offered by local 
liquor stores, and the availability of alcohol- 
and other drug-related paraphernalia in campus 
and community stores. 

Selecting a Research Design 

A research design is the basic structure 
of the evaluation, which outlines when and 
where data will be collected to study the effect of 
the environmental management program. 

Typically, an evaluation plan will call for 
data collection both before (pretest, or baseline) 
and after (posttest) the prevention program is 
launched. Positive changes in the intermediate 
and long-term outcomes might be taken as a 
sign that the program has been successful. 

The problem with this simple one group 
pretest-posttest design is that there are many 
other possible explanations for the obseived 
changes. For example, there might have been 
other changes occurring at the community or 
state level that had no direct connection with 


the intervention yet had the potential to 
influence student behavior. Or perhaps campus 
police, as they became more experienced with a 
new recordkeeping form, were more likely to note 
whether a reported incident involved student 
alcohol use. 

The puipose of more advanced research 
designs — involving data collection at nonin- 
tervention sites — is to increase confidence that 
any outcome changes that coincide with the 
program can actually be attributed to it. 

The most common design is a nonequiva- 
lent comparison group design, which involves 
pretest and posttest data collection at one or 
more sites with the program (“treatment 
group” or “experimental group”) and at one or 
more sites without the program (“comparison 
group” or “control group”). Building in this 
comparison makes it possible to rule out many 
competing explanations for any pretest-to- 
posttest changes in a given treatment group. 

Using an advanced design is less important if 
the environmental management strategies 
being used are already known to be effective — 
if not with college students then with the 
general population — for example, responsible 
beverage service, higher alcohol prices, increased 
enforcement of the minimum- age drinking law, 
or increased DUI enforcement. 14 

If that is not the case, then whether the 
research design can be expanded to include 
comparison sites will depend on funding avail- 
ability and the cooperation of officials from a 
comparable campus community, who need to 
agree to continue holding back on implementing 
environmental management strategies. This is not 
always feasible. 

In most cases, a more reasonable alternative 
might be to examine how any obseived changes 
in long-term outcomes compare with those 
being seen statewide or nationally at other 
campus communities. At the very least, campus 
administrators need to know whether their key 
indicators are moving in the right direction. 

Utilizing the Results 

The main reason for conducting an 
evaluation is to find out if a prevention 
program is working as intended. Process 
measures are used to document how the 
program was implemented, while outcome 



measures are used to establish the program’s 
effect on key environmental factors that affect 
student behavior, actual levels of students’ 
substance use, and the consequences of that use. 

The combined use of process and outcome 
measures can help administrators develop plans 
for strengthening or improving the program. If 
an environmental approach appears to be 
successful, then consider broadening its scope 
or investing additional resources. If a particular 
approach appears to have failed, then diagnose 
what went wrong. Was the program imple- 
mented as planned? If not, can that be 
corrected? Might the approach have succeeded if 
even more resources were put into it? Or does it 
need to be abandoned in favor of a new idea? 

Stated simply, evaluation is a management 
tool. Understanding the findings, and then 
using them to make program changes, requires 
the same deliberateness and care that originally 
went into developing, implementing, and evalu- 
ating the prevention program. 

William Dejong, Ph.D., is a professor of social 
and behavioral sciences at the Boston 
University School of Public Health and a 
senior Center adviser to the Higher Education 
Center for Alcohol and Other Drug Abuse and 
Violence Prevention. Linda M. Langford, 

Sc.D., is associate director for violence 
prevention initiatives at the Higher Education 
Center for Alcohol and Other Drug Abuse and 
Violence Prevention. 
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FIGURE 1. Responsible beverage service (RBS) logic model 
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Source: This logic model was developed by W. Dejong and L. Langford, Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention, 2005. 
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Resources 

The U.S. Department of Education’s Higher 
Education Center for Alcohol and Other Drug 
Abuse and Violence Prevention considers evalu- 
ation of programs and policies to be an impor- 
tant component of a comprehensive prevention 
approach. To help campus administrators 
develop and execute evaluations of prevention 
programs, the Higher Education Center offers 
publications and other materials for free on its 
Web site at www.higheredcenter.org. 

Publications 

College Alcohol Risk Assessment Guide: 
Environmental Approaches to Prevention 
by B. E. Ryan; T. Colthurst; and L. Segars 
This guide outlines methods for identifying and 
analyzing factors in the campus and commu- 
nity environment that contribute to alcohol- 
related problems. Scanning and analysis 
exercises are provided. 

Finding Out What Works and Why: A Guide 
to Evaluating College Prevention Programs 
and Policies 

by G. Dowdall; W. Dejong; and S. B. Austin 
Written for campus administrators, this intro- 
ductory guide outlines procedures for each step 
of the evaluation process: (1) describing the 
prevention program; (2) framing evaluation 
questions; (3) selecting measures; (4) choosing 
an evaluation design; and (5) reporting results. 

Ho w to Select a Program Evalua tor 
by L. Langford and W. Dejong 
This document describes the skills, expertise, 
and experience to look for when seeking an 
evaluator; questions to ask when considering an 
evaluator; and guidance on how to network to 
find the right person and forge an effective 
working relationship. 

Methods for Assessing Student Use of Alcohol 

and Other Drugs 

by W. Dejong and H. Wechsler 

This guide describes methods for gathering and 

interpreting student survey data on AOD-related 

problems. Methods for developing questions, 

drawing a random sample of students, and 

achieving high response rates are outlined. 


Selecting the Right Tool: A Compendium of Alcohol 
ami Other Drug Assessment ami Evaluation 
Instruments for Use in Higher Education 
by C. Presley; S. B. Austin; and J. Jacobs 
This compendium reviews issues to be considered 
when selecting data collection instruments for 
assessing campus-based prevention programs and 
describes leading instruments available to tire field. 
More than 20 instrument samples are included. 

Web Links to Other Materials 

The Higher Education Center’s Web site includes 
links to a rich compilation of evaluation resources 
organized by topic, including the following: 

• Evaluation readiness and building capacity 
for evaluation 

• Needs assessment and problem analysis 

• Program planning, with evaluation in mind 

• Evaluation design 

• Protection of human subjects 

• Measurement and data collection 

• Qualitative methods, including interviews, 
focus groups, and observation 

• Suiveys 

• Geographic Information Systems (GIS) 
mapping 

• Archival data 

• Data analysis and statistics 

• Reporting and publicizing findings 

Other Services 

The Higher Education Center provides training, 
infonnation, and technical assistance related to 
evaluation. For further information on these 
services, visit www.higheredcenter.org. 
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